When running a browsertime test with --browsertime-visualmetrics, the dependencies should be installed but appear to be failing often.

The outcome on linux (resolved):

• ffmpeg is listed as FAIL while other dependencies are OK

ran the following, and all pass, but when running the test again, ffmpeg fails again

• ./mach browsertime --clobber --setup --install-vismet-reqs
• Tried running mach clobber and mach build and repeating above steps, outcome is the same
• per :sparky 's suggestion I downloaded ffmpeg and added it to PATH: https://github.com/ncalexan/geckodriver/releases/download/v0.24.0-android/ffmpeg-4.1.4-i686-static.tar.xz
• was able to run visual metrics test without additional steps
• see: https://searchfox.org/mozilla-central/source/testing/raptor/mach_commands.py#188-190

Outcome on osx (unresolved):

• convert and compare FAIL

ran the following but made no impact, convert and compare still fail:

• ./mach browsertime --clobber --setup --install-vismet-reqs
• Tried running mach clobber and mach build and repeating above steps, outcome is the same

Will run on Windows as well to see what the outcome is

We should get this into a JIRA task and schedule it for next sprint - I think it's a big hole we're missing right now in raptor-browsertime.

On windows, when attempting to run test with visual metrics:

• imagemagick download URL gives 404, fix for this: https://bugzilla.mozilla.org/show_bug.cgi?id=1735540
• after updating url:

Missing browsertime files...attempting to install
OSError(8, '%1 is not a valid Win32 application', None, 193, None)

see https://searchfox.org/mozilla-central/source/testing/raptor/mach_commands.py#258

• tried running a test, ffmpeg convert and compare FAIL
• ran mach browsertime --clobber --setup --install-vismet-reqs
• dependencies install correctly
• tried running a test, ffmpeg convert and compare FAIL again
• downloaded ffmpeg and added to PATH, currently this does not appear to be working, still investigating

(unresolved)

(In reply to Kimberly Sereduck :kimberlythegeek from comment #0)

When running a browsertime test with --browsertime-visualmetrics, the dependencies should be installed but appear to be failing often.

The outcome on linux (resolved):

• ffmpeg is listed as FAIL while other dependencies are OK

ran the following, and all pass, but when running the test again, ffmpeg fails again

• ./mach browsertime --clobber --setup --install-vismet-reqs

It seems like this step needs to be repeated whenever Firefox is rebuilt

After Bug 1677559 is merged to m-c, this should be pretty much resolved (incl window and linux). I will do another pass and clean up and residual comments and double check the rst files before closing this

With Bug 176437 complete this meta issue is resolved